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DESCRIPTION 

IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM 
FOR IMAGE PROCESSING METHOD, AND RECORDING MEDIUM RECORDING 
THEREON PROGRAM FOR IMAGE PROCESSING METHOD 

5 

Technical Field 
The present invention relates to image progressing 
apparatuses, image processing methods, programs for image 
processing methods, and recording media recording thereon 
10 programs for image processing methods, and is applicable, 
for example, to resolution conversion. The present 
invention is capable of effectively avoiding loss of high- 
frequency components and preventing occurrence of j aggies by 
detecting an edge gradient direction with the largest 
^ 15 gradient of pixel values and an edge direction orthogonal to 

the edge gradient direction and by performing edge 
enhancement and smoothing processing in the edge gradient 
direction and in the edge direction, respectively, to 
generate output image data. 

20 

BACKGROUND OF THE INVENTION 
Background Art 
In image processing, resolution conversion has been 
performed, for example, by linear interpolation or bicubic 
2 5 conversion. A method for performing such image processing 
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with a simplified configuration is proposed, for example, in 
Japanese Unexamined Patent Application Publication No. 2003- 
224715 . 

However, in such image processing, j aggies occur in 
5 edge portions. When a characteristic of linear 

interpolation or bicubic conversion is set such that j aggies 
do not stand out, there is a problem that high-frequency 
components of an image are lost and that the image is 
blurred due to a reduction in the sharpness. Since 
10 considerable image blur occurs in texture portions, the 
characteristic of such processing may be changed in the 
texture portions. However, in this case, heterogeneity and 
flickering occur in positions where the characteristic is 
changed. 

15 

Disclosure of Invention 
The present invention is designed in view of the 
foregoing points, and proposes an image processing apparatus, 
an image processing method, a program for an image 
2 0 processing method, and a recording medium recording thereon 
a program for an image processing method that are capable of 
effectively avoiding loss of high-frequency components and 
preventing occurrence of j aggies. 

In order to solve the above problems, the present 
2 5 invention is applied to an image processing apparatus for 
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processing input image data and for outputting output image 
data. The image processing apparatus includes an edge 
detection unit for detecting an edge gradient direction with 
the largest gradient of pixel values and an edge direction 
5 orthogonal to the edge gradient direction for each pixel of 
the input image data; an edge direction processing unit for 
performing smoothing processing on the image data in the 
edge direction for each pixel of the output image data in 
accordance with a detection result of the edge detection 

10 unit and for sequentially detecting pixel values 

corresponding to respective pixels of the output image data; 
and an edge gradient direction processing unit for 
performing edge enhancement processing in the edge gradient 
direction on the pixel values output from the edge direction 

15 processing unit for the respective pixels of the output 

image data in accordance with the detection result of the 
edge detection unit and for sequentially outputting pixel 
values of the output image data. 

The configuration according to the present invention is 

2 0 applied to an image processing apparatus for processing 

input image data and for outputting output image data. The 
image processing apparatus includes an edge detection unit 
for detecting an edge gradient direction with the largest 
gradient of pixel values and an edge direction orthogonal to 

2 5 the edge gradient direction for each pixel of the input 
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image data; an edge direction processing unit for performing 
smoothing processing on the image data in the edge direction 
for each pixel of the output image data in accordance with a 
detection result of the edge detection unit and for 
5 sequentially detecting pixel values corresponding to 

respective pixels of the output image data; and an edge 
gradient direction processing unit for performing edge 
enhancement processing in the edge gradient direction on the 
pixel values output from the edge direction processing unit 

10 for the respective pixels of the output image data in 

accordance with the detection result of the edge detection 
unit and for sequentially outputting pixel values of the 
output image data. Thus, edge enhancement and smoothing 
processing are performed in the edge gradient direction with 

15 the largest gradient of pixel values and in the edge 
direction orthogonal to the edge gradient direction, 
respectively, to generate the output image data. Therefore, 
loss of high-frequency components can be effectively avoided, 
and occurrence of j aggies can be prevented. 

20 In addition, the present invention is applied to an 

image processing method for processing input image data and 
for outputting output image data. The image processing 
method includes an edge detection step of detecting an edge 
gradient direction with the largest gradient of pixel values 

2 5 and an edge direction orthogonal to the edge gradient 
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direction for each pixel of the input image data; an edge 
direction processing step of performing smoothing processing 
on the image data in the edge direction for each pixel of 
the output image data in accordance with a detection result 
5 by the edge detection step and sequentially detecting pixel 
values corresponding to respective pixels of the output 
image data; and an edge gradient direction processing step 
of performing edge enhancement processing in the edge 
gradient direction on the pixel values detected by the edge 

10 direction processing step for the respective pixels of the 

output image data in accordance with the detection result by 
the edge detection step and sequentially outputting pixel 
values of the output image data. 

Thus, with the configuration according to the present 

15 invention, an image processing method capable of effectively 
avoiding loss of high-frequency components and preventing 
occurrence of j aggies can be provided. 

In addition, the present invention is applied to a 
program for an image processing method performed by 

2 0 arithmetic processing means for processing input image data 
and for outputting output image data. The program includes 
an edge detection step of detecting an edge gradient 
direction with the largest gradient of pixel values and an 
edge direction orthogonal to the edge gradient direction for 

25 each pixel of the input image data; an edge direction 
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processing step of performing smoothing processing on the 
image data in the edge direction for each pixel of the 
output image data in accordance with a detection result by 
the edge detection step and sequentially detecting pixel 
5 values corresponding to respective pixels of the output 

image data; and an edge gradient direction processing step 
of performing edge enhancement processing in the edge 
gradient direction on the pixel values detected by the edge 
direction processing step for the respective pixels of the 

10 output image data in accordance with the detection result by 
the edge detection step and sequentially outputting pixel 
values of the output image data. 

Thus, with the configuration according to the present 
invention, a program for an image processing method capable 

15 of effectively avoiding loss of high-frequency components 
and preventing occurrence of j aggies can be provided. 

In addition, the present invention is applied to a 
recording medium recording thereon a program for an image 
processing method performed by arithmetic processing means 

2 0 for processing input image data and for outputting output 
image data. The program for the image processing method 
includes an edge detection step of detecting an edge 
gradient direction with the largest gradient of pixel values 
and an edge direction orthogonal to the edge gradient 

2 5 direction for each pixel of the input image data; an edge 
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direction processing step of performing smoothing processing 
on the image data in the edge direction for each pixel of 
the output image data in accordance with a detection result 
by the edge detection step and sequentially detecting pixel 
values corresponding to respective pixels of the output 
image data; and an edge gradient direction processing step 
of performing edge enhancement processing in the edge 
gradient direction on the pixel values detected by the edge 
direction processing step for the respective pixels of the 
output image data in accordance with the detection result by 
the edge detection step and sequentially outputting pixel 
values of the output image data. 

Thus, with the configuration according to the present 
invention, a recording medium recording thereon a program 
for an image processing method capable of effectively 
avoiding loss of high-frequency components and preventing 
occurrence of j aggies can be provided. 

According to the present invention, loss of high- 
frequency components can be effectively avoided and 
occurrence of j aggies can be prevented. 

Brief Description of the Drawings 
Fig. 1 is a functional block diagram showing the 
configuration of an image processing apparatus according to 
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an embodiment of the present invention. 

Fig. 2 is a schematic diagram for generation of a pixel 
gradient matrix. 

Fig. 3 is a schematic diagram for explaining an edge 
5 gradient direction and an edge direction. 

Fig. 4 is a schematic diagram for explaining an 
operation of an edge direction processing unit. 

Fig. 5 is a characteristic curve diagram showing a 
parameter used for setting a smoothing processing range. 
10 Fig. 6 is a characteristic curve diagram showing 

another parameter used for setting the smoothing processing 
range . 

Fig. 7 is a schematic diagram used for explaining an 
operation of an edge gradient direction processing unit. 
15 Fig. 8 is a characteristic curve diagram showing a 

parameter used for setting a blend processing unit. 

Fig. 9 is a characteristic curve diagram showing 
another parameter used for setting the blend processing unit. 



2 0 Best Mode for Carrying Out the Invention 

Embodiments of the present invention will be described 
by appropriately referring to the drawings . 
(1) Configuration of Embodiment 

Fig. 1 is a functional block diagram showing an image 
25 processing apparatus according to an embodiment of the 
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present invention. The image processing apparatus 1 is 
formed by, for example, a digital signal processor 
functioning as arithmetic processing means. By a 
predetermined processing program performed by the arithmetic 
5 processing means, the resolution of image data Dl, which is 
input image data, is converted into a resolution designated 
by a controller, which is not shown, and the converted image 
data is output. Accordingly, the image processing apparatus 
1 generates output image data D2 obtained by magnifying or 
10 reducing an image based on the image data Dl, and outputs 
the generated output image data D2 to displaying means or 
the like. 

Although the processing program for the arithmetic 
processing means is supplied by being pre- installed in the 

15 image processing apparatus in this embodiment, for example, 
such a processing program may be supplied by being 
downloaded via a network, such as the Internet, or may be 
supplied via various recording media. Such recording media 
may be widely applicable to various recording media, such as 

2 0 optical disks, memory cards, and detachable hard disk drives. 

In the image processing apparatus 1, an edge detection 
unit 2 detects an edge gradient direction with the largest 
gradient of pixel values and an edge direction orthogonal to 
the edge gradient direction for each pixel of the image data 

25 Dl . In other words, in the edge detection unit 2, a 
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gradient matrix generation unit 3 sequentially selects a 
target pixel, for example, in a raster scanning order, and, 
as shown in Fig. 2, generates a brightness gradient matrix G 
represented by the next condition by performing arithmetic 
processing using pixel values in a range W centered on the 
tart pixel. Fig. 2 shows an example in which ±3 pixels in x 
and y directions centered on a target pixel are set as the 
range W. 



G = f g g T w d A 
J w 
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In addition, "w (i '3 ) " represents a Gaussian weight 
represented by condition (2) , and "g M represents a 
brightness gradient represented by condition (3) using a 
partial differential "gx" in the x direction of an image 
brightness I and a partial differential "gy" in the y 
direction of the image brightness I. 
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Accordingly, in the edge detection unit 2, the gradient 
matrix generation unit 3 detects a brightness gradient 
obtained by weighting based on a target pixel in the 
5 predetermined range W centered on the target pixel. 

In the edge detection unit 2, a subsequent eigenvalue 
and eigenvector detection unit 4 detects an edge gradient 
direction vl with the largest gradient of pixel values and 
an edge direction v2 orthogonal to the edge gradient 
10 direction vl for a target pixel, as shown in Fig. 3, by 

processing the brightness gradient matrix G generated by the 
gradient matrix generation unit 3. In addition, the 
eigenvalue and eigenvector detection unit 4 detects 
eigenvalues Xl and X2 representing dispersions of gradients 
15 of the pixel values in the edge gradient direction vl and in 
the edge direction v2 , respectively. 

More specifically, the eigenvalue and eigenvector 
detection unit 4 detects the edge gradient direction vl, the 
edge direction v2 , and the eigenvalues Xl and X2 (Xl > X2 ) in 
20 accordance with the following arithmetic processing: 



A1 = 



2 (4) 
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v1 = 



v'1 



v'1 = 



^x x ^yy *f 

2gT~ 



, 1 



v2 =• 



v'2 



v'2 



(6) 



v'2 = 



( G — G —J" 

x x yy 

2 G_ 



(7) , 



where "a" is defined by the following condition: 

(8) . 



a=G 2 + 4G 2 - 2G G +G 2 

X X X y x x^y y 1 w yy 



An edge direction processing unit 5 calculates an edge 
direction vc for a pixel of the image data D2 acquired after 
resolution conversion in accordance with the edge direction 

10 v2 for a target pixel of the image data Dl detected as 
described above by the edge detection unit 2, and 
sequentially calculates a pixel value corresponding to each 
pixel Pc of the output image data D2 by interpolation 
processing based on the edge direction vc . 

15 In other words, as shown in Fig. 4, the edge direction 

processing unit 5 performs arithmetic processing based on 
the next condition for pixels of the image data Dl (in the 
example shown in Fig. 4, P3 , P4 , P9, and P10) adjacent to a 
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pixel Pc of the output image data D2 to be calculated 
(hereinafter, referred to as a target pixel of the image 
data D2) , and calculates the edge direction vc of the target 
pixel Pc by performing interpolation processing using the 
5 edge directions v2 (v3, v4, v9, and vlO) of the adjacent 
pixels of the image data Dl . 

V C =0 " ty XO " *x )V 3 + t x V 4 >^ t y ((l " t x )V 9 + t x V 10 ) 

(9) 

Here, "t x " and "t y n represent coordinate values of a 
10 target pixel for internally dividing the sampling points P3 , 
P4, P9, and P10 based on the image data Dl in the x and y 
directions, and the conditions 0 < t x < 1 and 0 < t y < 1 are 
satisfied. 

In addition, in accordance with the edge direction vc 
15 for the target pixel Pc calculated as described above, the 

edge direction processing unit 5 sets a predetermined number 
of sampling points P-2, P-l, PI, and P2 based on a sampling 
pitch in the image data Dl from the sampling points for the 
target pixel Pc on a line in the edge direction vc . In 
20 addition, pixel values of the sampling points P-2, P-l, PI, 
and P2 and the target pixel Pc are calculated by an 
interpolation operation using pixel values of the image data 
Dl . Accordingly, in accordance with detection results of 
the edge detection unit 2 , interpolated image data in the 
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edge direction based on interpolation processing for the 
input image data Dl is generated on the line extending in 
the edge direction vc for each pixel of the output image 
data D2 . 

5 In addition, in accordance with a calculation result of 

an edge direction range determination unit 6, which will be 
described below, the number of sampling points set as 
described above is changed, and the subsequent filtering 
processing is changed. Thus, the number of taps for the 

10 filtering processing is changed in accordance with 

reliability of an edge in the edge direction vc of the 
target pixel. More specifically, for example, in a case 
where the subsequent filtering processing is performed based 
on 3-tap filtering, a pixel value of the target pixel Pc is 

15 calculated by linear interpolation using the peripheral 

pixels P3, P4, P9, and P10, and pixel values of the previous 
and subsequent sampling points P-l and PI are calculated by 
linear interpolation using P2 , P3 , P8, and P9; and P4 , P5 , 
P10, and Pll, respectively. In contrast, in a case where 

2 0 the subsequent filtering processing is performed based on 5- 
tap filtering, a pixel value of the target pixel Pc is 
calculated by linear interpolation using the peripheral 
pixels P3, P4, P9, and P10, and pixel values of the sampling 
points P-2, P-l, PI, and P2 are calculated in a similar way. 

25 Then, the edge direction processing unit 5 smoothes the 
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calculated pixel values of the sampling points P-2, P-l, PI, 
and P2 and the target pixel Pc by filtering processing, and 
determines a pixel value Pc 1 of the target pixel Pc. In 
other words, for 3 -tap filtering, for example, the pixel 
5 value PC of the target pixel Pc is calculated by arithmetic 
processing represented by the following condition: 

P' c = 0.2 5xP_ ,+ 0. 5xP c + 0.2 5xP +1 

In contrast, for 5- tap filtering, for example, the 
pixel value Pc 1 of the target pixel Pc is calculated by 
10 arithmetic processing represented by the following 
condition : 

P' c =0.06 2 5xP_ 2 +0.25xP_ 1 + 0.37 5xP c 

+ 0.2 5xP + 1 +0.0625xP +2 

(11). 

As described above, in this embodiment, a pixel value 
15 corresponding to a pixel of the output image data D2 is 
calculated by performing smoothing processing on 
interpolated image data in an edge direction. Thus, loss of 
high-frequency components can be effectively avoided and 
occurrence of j aggies in an edge can be prevented. In 
2 0 addition, an interpolation operation for generating such 
interpolated image data need not be linear interpolation 
using pixel values of adjacent neighboring pixels, and 
various interpolation operation methods using various 
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peripheral pixels are widely applicable. In addition, 
arithmetic processing for filtering processing using the 
interpolated image data need not be the arithmetic 
processing based on condition (10) or (11) , and 
5 interpolation operations using various weighting 
coefficients are widely applicable. 

In a case where pixel values in the image data D2 are 
calculated by detecting edge directions for respective 
pixels as described above, smoothing processing may be 

10 performed in a direction orthogonal to a brightness gradient 
in a portion other than an edge. In such a case, if 
filtering processing is performed with a large number of 
taps involving a wide range, an image quality is reduced. 
In contrast, however, in an edge portion, filtering 

15 processing performed in a wide range prevents occurrence of 
j aggies more reliably and thus forms a smooth edge. 

Thus, in this embodiment, the number of taps for 
filtering processing is changed for each pixel. Thus, a 
range for smoothing processing in an edge direction is 

20 changed for each pixel. In addition, such a filtering range 
is changed in accordance with reliability of an edge in the 
edge direction vc. Therefore, a reduction in the image 
quality due to smoothing processing can be prevented. 

More specifically, in this embodiment, in accordance 

2 5 with the ratio X2/X1 of the eigenvalue X2 for the edge 
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direction v2 to the eigenvalue Xl for the edge gradient 
direction vl, reliability of an edge in the edge direction 
vc is detected. In other words, if the ratio X2/X1 is small, 
it is determined that the pixel value gradient in the edge 
5 gradient direction vl is predominant in this target pixel 
and that a stronger edge appears in the edge direction v2 . 
Thus, as shown in Fig. 5, the edge direction processing 
range determination unit 6 generates a parameter p that 
increases substantially linearly in accordance with a 

10 decrease in the value of the ratio X2/X1 when the ratio 

X2/X1 is within a predetermined range between A,2/^lmin and 
X,2/Xlmax and that exhibits the maximum value pmax or the 
minimum value pmin when the ratio X2/X1 is outside the 
predetermined range between ^2/Xlmin and ^2/Xlmax. 

15 Accordingly, the parameter p changing depending on 

reliability of an edge in an edge direction is generated. 

If the eigenvalue Xl for the edge gradient direction vl 
is large, the contrast across the edge is large. Thus, this 
edge is regarded as being a distinctive edge. Thus, as 

2 0 shown in Fig. 6, the edge direction processing range 

determination unit 6 generates a parameter q that increases 
substantially linearly in accordance with the eigenvalue Xl 
in a predetermined range between Xlmin and Xlmax and that 
exhibits a lower limit value qmin or an upper limit value 

2 5 qmax in a range smaller or larger than the range between A,l 
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min and Xlmax. Accordingly, the parameter q changing in 
accordance with rising of an edge is generated. 

The edge direction processing range determination unit 
6 performs multiplication represented by the next condition 
5 for the two parameters p and q to calculate a filtering 
range r for the edge direction processing. 



In order to correspond to the sampling points of the 
image data D2 in the processing of the edge direction 

10 processing unit 5, the edge direction processing range 
determination unit 6 converts the eigenvalues Xl and X2 
based on the sampling points of the image data Dl into 
eigenvalues based on the sampling points of the image data 
D2, and calculates a filtering range r. In this case, after 

15 a filtering range r is calculated from the sampling points 
of the image data Dl, a filtering range r based on the 
sampling points of the image data D2 may be calculated by 
interpolation processing for the calculation result. In 
contrast, after the eigenvalues Xl and X2 based on the 

2 0 sampling points of the image data D2 are calculated by 
interpolating the eigenvalues Xl and X2 based on the 
sampling points of the image data Dl, a filtering range r 
based on the sampling points of the image data D2 may be 
calculated from the calculation result. 

25 Accordingly, the edge direction processing unit 5 
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changes the number of taps for the filtering processing in 
accordance with the range r calculated as described above, 
and calculates the pixel value Pc 1 of the target pixel Pc of 
the image data D2 . 
5 In such filtering processing, the edge direction 

processing unit 5 performs filtering processing based on the 
real number of taps by blending filtering processing results. 
Thus, the edge direction processing unit 5 overcomes 
unnaturalness in changing the number of taps when the 

10 filtering processing is performed based on the integral 
number of taps . 

In other words, in the edge direction processing unit 5, 
a filter with the integral number of taps represented by the 
next condition is defined. Here, an odd number, such as 1, 

15 3, 5, , is applied as the integral number of taps. 

"inl.g.r = f !<>Or ( 11,.., ) 
n iVta«er =Ceil ( n rea| ) 

(13) 

Here, "floor(n) " represents the maximum integral number 
of taps not exceeding n, and "ceil(n)" represents the 
2 0 minimum integral number of taps larger than n. In addition, 
the range r calculated based on condition (12) is applied to 
"nreal". Thus, when n is 3.5, the number of taps 
"floor (n) "is 3, and the number of taps "ceil (n) " is 5. 

Blending of the filtering processing results is 
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performed by calculating a filtering processing result f (n) 
based on a real number by performing arithmetic processing 
represented by the next condition using the two types of 
filtering processing results. Thus, the edge direction 
5 processing unit 5 calculates the pixel value Pc 1 of the 

target pixel Pc of the image data D2 by performing filtering 
processing based on the two types of tap numbers in the 
filtering range r and by further performing arithmetic 
processing represented by condition (14) using the two types 

10 of filtering processing results. Accordingly, the edge 

direction processing unit 5 calculates the pixel value Pc 1 
of the target pixel Pc of the image data D2 by performing 
filtering processing based on the number of taps 
corresponding to reliability of an edge in an edge direction, 

15 and changes the number of taps in a decimal fractional part. 

f(n) = af (n- 1 teger ) + (!-<* )f (n+UJ (i4) 

here, ** = n ' oal ) / 2 

An edge gradient direction processing unit 7 performs 
edge enhancement processing in the edge gradient direction 
2 0 vl using the pixel value Pc ' of the target pixel Pc of the 
image data D2 calculated as described above by the edge 
direction processing unit 5. In other words, as shown in 
Fig. 7, the edge gradient direction processing unit 7 
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calculates an edge gradient direction vg for a target pixel 
Pec of the image data D2 in accordance with the edge 
gradient direction vl based on adjacent sampling points of 
the image data Dl , as in calculation of the edge direction 
5 vc for the target pixel Pc of the image data D2 performed by 
the edge direction processing unit 5. 

In addition, in accordance with the edge gradient 
direction vg for the target pixel Pec calculated as 
described above, the edge gradient direction processing unit 

10 7 sets a predetermined number of sampling points Pc-1 and 

Pc + 1 based on a sampling pitch in the image data D2 from the 
sampling points for the target pixel Pec on a line in the 
edge gradient direction vg. In addition, the edge gradient 
direction processing unit 7 calculates pixel values of the 

15 sampling points Pc-1 and Pc+1 and the target pixel Pec by an 
interpolation operation using pixel values output from the 
edge direction processing unit 5 . Accordingly, in 
accordance with detection results of the edge detection unit 
2, the edge gradient direction processing unit 7 generates 

2 0 interpolated image data in the edge gradient direction 
obtained by interpolation processing for the image data 
based on the pixel values output from the edge direction 
processing unit 5 on a line extending in the edge gradient 
direction vg for each pixel of the output image data D2 . 

2 5 Then, the edge gradient direction processing unit 7 
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performs filtering processing on the pixel values of the 
sampling points Pc-1 and PC+1 and the target pixel Pec 
calculated as described above, and determines a pixel value 
PcC of the target pixel Pec. In the example shown in Fig. 
5 7, a case where the pixel value Pec 1 of the target pixel Pec 
is calculated using three taps is described. The pixel 
value of the sampling point Pc-1 is generated by linear 
interpolation based on peripheral sampling points Pel, Pc2, 
Pc4, and Pec, and the pixel value of the sampling point Pc+1 

10 is generated by linear interpolation based on peripheral 
sampling points Pec, Pc5, Pc7, and Pc8 . Accordingly, the 
edge gradient direction processing unit 7 performs edge 
enhancement in the direction that crosses the edge. In such 
an interpolation operation for generating interpolated image 

15 data, linear interpolation using pixel values of adjacent 

neighboring pixels need not be performed, and interpolation 
operation methods using various peripheral pixels are widely 
applicable. In addition, as arithmetic processing for 
filtering processing using such interpolated image data, 

20 interpolation operations using various weighting 
coefficients are widely applicable. 

An interpolation processing unit 8 converts a 
resolution of the image data Dl, for example, by linear 
interpolation or bicubic conversion, and outputs a pixel 

25 value Pa based on a sampling pitch corresponding to the 
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image data D2 . 

A blend ratio determination unit 9 generates a 
weighting coefficient for blending in accordance with 
reliability of an edge in the edge direction vc. In other 
5 words, when smoothing processing is performed in the edge 
direction and edge enhancement is performed in a direction 
orthogonal to the edge, unnatural edge enhancement may occur 
in a natural image. Thus, in this embodiment, a blend 
processing unit 10 performs weighting addition of the pixel 

10 value Pa of the image data D2 generated by the interpolation 
processing unit 8 using a known procedure and the pixel 
value Pec' generated by the edge gradient direction 
processing unit 7 in order to generate the image data D2 , 
and the blend ratio determination unit 9 changes the 

15 weighting coefficient for the weighting addition processing. 
In addition, the blend ratio determination unit 9 changes 
the weighting coefficient in accordance with reliability of 
an edge in an edge direction, and this prevents excessive 
unnaturalness in processing for the edge. In addition, the 

20 ratio X2/X1 of the eigenvalue X2 for the edge direction v2 
to the eigenvalue Xl for the edge gradient direction vl is 
applied to the reliability of the edge in the edge direction. 

More specifically, if the ratio X2/X.1 is small, it is 
determined that the gradient in the edge gradient direction 

25 vl is predominant in this target pixel and that a stronger 
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edge appears in the edge direction v2 . Thus, as shown in 
Fig. 8, the blend ratio determination unit 9 generates a 
parameter s that increases substantially linearly in 
accordance with a decrease in the value of the ratio X2/X1 
5 when the ratio X2/X1 is within a predetermined range between 
A,2/^lmin and A,2/A,lmax and that exhibits the maximum value 
smax or the minimum value smin when the ratio X2/X1 is 
outside the predetermined range between ^2/A,min and A,2/tanax. 
Accordingly, the parameter s changing depending on 

10 reliability of an edge in an edge direction is generated. 

If the eigenvalue Xl for the edge gradient direction vl 
is large, the contrast across the edge is large. Thus, this 
edge is regarded as being a distinctive edge. Thus, as 
shown in Fig. 9, the blend ratio determination unit 9 

15 generates a parameter t that increases substantially 
linearly in accordance with the eigenvalue Xl in a 
predetermined range between A,lmin and ?ilmax and that 
exhibits a lower limit value tmin or an upper limit value 
tmax in a range outside the range between ^lmin and Xlmax. 

2 0 Accordingly, the parameter t changing in accordance with 
rising of an edge is generated. 

The blend ratio determination unit 9 performs 
multiplication represented by the next condition for the two 
parameters s and t to calculate a weighting coefficient p 

25 for blending (0 < P < 1) . 
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In order to correspond to the sampling points of the 
image data D2 , the blend ratio determination unit 9 converts 
the eigenvalues A.1 and X2 based on the sampling points of 
5 the image data Dl into eigenvalues based on the sampling 
points of the image data D2 , and calculates a weighting 
coefficient a for blending. In this case, after a weighting 
coefficient a for blending is calculated from the sampling 
points of the image data Dl, a weighting coefficient a for 

10 blending based on the sampling points of the image data D2 
may be calculated by interpolation processing for the 
calculation result. In contrast, after the eigenvalues A,l 
and X2 based on the sampling points of the image data D2 are 
calculated by interpolating the eigenvalues ^,1 and X2 based 

15 on the sampling points of the image data Dl, a weighting 

coefficient P for blending based on the sampling points of 
the image data D2 may be calculated from the calculation 
result . 

The blend processing unit 10 performs weighting 
20 addition of image data S3 based on the pixel value Pec 1 

calculated by the edge gradient direction processing unit 7 
and image data Sll based on the pixel value Pa calculated by 
the interpolation processing unit 8 using the weighting 
coefficient P obtained by the blend ratio determination unit 
2 5 9 by performing arithmetic processing represented by the 
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next condition, and outputs the processing result as the 
image data D2 . 

S4 = /?xS3 + (l - £)x S11 . _ 

(Id) 

(2) Operation of Embodiment 

With the foregoing configuration, the input image data 
Dl (Fig. 1) is input to the edge detection unit 2. In the 
edge detection unit 2, the edge gradient direction vl with 
the largest gradient of pixel values and the edge direction 
v2 orthogonal to the edge gradient direction vl are 
sequentially detected for each pixel (Figs. 2 and 3) . In 
addition, the input image data Dl is input to the edge 
direction processing unit 5 . In the edge direction 
processing unit 5, the image data is smoothed in the edge 
direction v2 for each pixel of the output image data D2 in 
accordance with a detection result of the edge detection 
unit 2, and a pixel value Pc corresponding to each pixel of 
the output image data D2 is sequentially calculated. In 
addition, the pixel value Pc is input to the edge gradient 
direction processing unit 7 in accordance with a calculation 
result. In the edge gradient direction processing unit 7, 
in accordance with the detection result of the edge 
detection unit 2, edge enhancement is performed in the edge 
gradient direction vl for each pixel of the output image 
data D2, and a pixel value of the output image data D2 is 
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calculated. Accordingly, since the input image data Dl is 
smoothed in the edge direction v2 , occurrence of j aggies can 
be effectively avoided. In addition, edge enhancement is 
performed in the edge gradient direction vl orthogonal to 
5 the edge direction v2 . Thus, high-frequency components are 
enhanced in the direction orthogonal to the edge direction. 
Therefore, conversion into the image data D2 is performed 
while effectively avoiding loss of high-frequency components 
and preventing occurrence of j aggies. 

10 By performing a series of processing as described above, 

concerning the input image data Dl, in the edge direction 
processing unit 5 (Fig. 4), interpolated image data Pc in 
the edge direction based on interpolation processing for the 
input image data Dl is generated for each pixel of the 

15 output image data D2 on a line extending in the edge 

direction v2 in accordance with a detection result of the 
edge detection unit 2, and the pixel value Pc 1 corresponding 
to each pixel of the output image data D2 is sequentially 
calculated by filtering processing of the interpolated image 

20 data Pc in the edge direction generated as described above. 
Accordingly, by setting a characteristic for the filtering 
processing, smoothing processing is performed in the edge 
direction v2 , and occurrence of j aggies can be effectively 
avoided . 

2 5 Concerning the input image data Dl, in the edge 
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detection unit 2, the gradient matrix generation unit 3 
generates a brightness gradient matrix G for each pixel, and 
the subsequent eigenvalue and eigenvector detection unit 4 
processes this brightness gradient matrix G to detect the 
5 edge gradient direction vl and the edge direction v2 . At 
the same time, the eigenvalues Xl and X2 representing 
dispersions of pixel value gradients for the edge gradient 
direction vl and the edge direction v2 are also calculated. 
From among calculation results of the edge detection 

10 unit 2, for the eigenvalues Xl and X2, the ratio X2/X1 of 

the eigenvalue X2 to the eigenvalue Xl is calculated by the 
edge direction processing range determination unit 6 . The 
ratio X2/X1 decreases as the pixel gradient in the edge 
gradient direction is more predominant. Accordingly, 

15 reliability of an edge is detected. Thus, in the edge 
direction processing range determination unit 6, the 
parameter p representing the reliability of the edge is 
generated in accordance with the ratio X2/X1 (Fig. 5) . Thus, 
in this embodiment, the reliability of the edge is 

20 calculated by effectively using the gradient matrix g used 
for calculation of the edge gradient direction vl and the 
edge direction v2 . In addition, since the eigenvalue Xl 
represents the strength of the edge, the parameter q 
corresponding to the parameter p is generated in accordance 

25 with the eigenvalue Xl (Fig. 6) . In addition, the filtering 
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range r is calculated by multiplication of the parameters p 
and q. 

Concerning the input image data Dl, when the edge 
direction processing unit 5 performs filtering processing on 
5 the interpolated image data Pc in the edge direction to 
generate the pixel value Pc 1 of the output image D2 , the 
number of taps for the filtering processing is changed in 
accordance with the filtering range r generated by the edge 
direction processing range determination unit 6 as described 

10 above. If the reliability of the edge is high, the pixel 
value Pc 1 of the output image data D2 is generated by 
performing filtering processing on the interpolated image 
data Pc in the edge direction in a wide range. If the 
reliability of the edge is low, the pixel value Pc ' of the 

15 output image data D2 is generated by performing filtering 
processing in a narrow range. Thus, for the input image 
data Dl, the number of taps for the filtering processing for 
the edge direction v2 is changed in accordance with the 
reliability of the edge. Thus, excessive smoothing 

20 processing in a portion other than the edge can be prevented, 
and a reduction in the image quality can be effectively 
avoided . 

In addition, by performing smoothing processing as 
described above, for the image data Dl, the weighting 
25 coefficient a for filtering processing is changed (condition 
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(12)) in accordance with the reliability of the edge in the 
edge direction v2 . In addition, by performing weighting 
addition of filtering processing results with different 
numbers of taps using the weighting coefficient a (condition 
5 (14) ) , the number of taps is changed in a decimal fractional 
part. Thus, unnaturalness in changing the integral number 
of taps is overcome. 

Concerning the image data obtained by calculation in 
the edge direction processing unit 5, in the edge gradient 

10 direction processing unit 7, in accordance with a detection 
result of the edge detection unit 2 , interpolated image data 
Pec in the edge gradient direction obtained by interpolation 
processing for the image data based on the pixel values 
output from the edge direction processing unit 5 is 

15 generated for each pixel of the output image data D2 on a 
line extending in the edge gradient direction vl (Fig. 7) , 
and the pixel value Pec' of the output image data D2 is 
calculated by performing filtering processing of the 
interpolated image data Pec in the edge gradient direction. 

20 Accordingly, by setting a characteristic for the filtering 
processing, edge enhancement is performed in the edge 
gradient direction vl , and the image data D2 is generated. 

For output data of the edge gradient direction 
processing unit 7 generated as described, however, when the 

25 image data Dl based on a natural image is processed, the 
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outline may be excessively enhanced. Thus, for the image 
data Dl, the interpolation processing unit 8 calculates a 
pixel value Pa' for each pixel of the output image data D2 
by interpolation processing in accordance with a known 
procedure. Accordingly, although high-frequency components 
of the image data Sll generated by the interpolation 
processing unit 8 based on the known procedure are lost, the 
outline of the image data Sll is not excessively enhanced. 

For the image data Dl, the blend processing unit 10 
performs weighting addition of the image data Sll from the 
interpolation processing unit 8 and the image data S3 from 
the edge gradient direction processing unit 7 in accordance 
with an image to generate the output image data D2 . Thus, a 
portion in which the outline is excessively enhanced is 
corrected using a pixel value Pa' in accordance with the 
known procedure, and the output image data D2 is generated. 
Therefore, in this embodiment, a reduction in the image 
quality due to excessive edge enhancement can be prevented. 

In this embodiment, similarly to a case where the 
filtering range r is calculated in the edge direction 
processing range determination unit 6, the reliability of an 
edge is calculated using the ratio X2/X1 of the eigenvalue 
X2 to the eigenvalue ^1, and the weighting coefficient P for 
weighting addition performed in the blend processing unit 10 
is calculated in accordance with the reliability of the edge 
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(Fig. 8) . In addition, the weighting coefficient p is 
corrected using the eigenvalue ^1 (Fig. 9) . Thus, in this 
embodiment, in such processing for preventing excessive edge 
enhancement, the gradient matrix g used for calculating the 
5 edge gradient direction vl and the edge direction v2 is 

effectively used to calculate the reliability of the edge, 
and the weighting coefficient P is set in accordance with 
the reliability of the edge. Thus, excessive edge 
enhancement can be prevented with such a simplified 
10 configuration. 

(3) Effects of Embodiment 

With the foregoing configuration, the edge gradient 
direction vl with the largest gradient of pixel values and 
the edge direction v2 orthogonal to the edge gradient 

15 direction vl are detected, and edge enhancement and 

smoothing processing are performed in the edge gradient 
direction vl and the edge direction v2 , respectively, to 
generate the output image data D2 . Thus, loss of high- 
frequency components can be effectively avoided and 

2 0 occurrence of j aggies can be prevented. 

In filtering processing for such smoothing processing, 
after the interpolated image data Pc in the edge direction 
obtained by interpolation processing for the input image 
data Dl is generated for each pixel of the output image data 

2 5 D2 on a line extending in the edge direction v2 in 
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accordance with a detection result of the edge detection 
unit 2, the pixel value Pc ' corresponding to each pixel of 
the output image data D2 is sequentially detected by 
performing filtering processing of the interpolated image 
5 data Pc in the edge direction. Accordingly, a 

characteristic for the filtering processing is variously set, 
and smoothing processing with a desired characteristic can 
thus be performed. 

In addition, the number of taps for the filtering 
10 processing is changed in accordance with the reliability of 
the edge in the edge direction v2 . Thus, incorrect 
smoothing processing in a portion other than the edge can be 
prevented . 

In addition, the weighting coefficient a for the 
15 filtering processing is changed in accordance with the 

reliability of the edge in the edge direction v2 , and the 
number of taps is changed in a decimal fractional part by 
performing weighting addition of filtering processing 
results of different numbers of taps using the weighting 
2 0 coefficient a for the filtering processing. Thus, by 
changing the number of taps as described above, 
unnaturalness in changing the integral number of taps can be 
effectively avoided. 

In addition, since the reliability of the edge in the 
25 edge direction v2 is the ratio of the dispersion X2 of the 
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pixel value gradient in the edge direction v2 to the 
dispersion Xl of the pixel value gradient in the edge 
gradient direction vl, the configuration for detecting the 
edge gradient direction vl and the edge direction v2 is 
5 effectively used, and unnaturalness in changing the number 
of taps can be effectively prevented. 

In contrast, after the interpolated image data Pec in 
the edge gradient direction obtained by interpolation 
processing for the image data Pc based on the pixel value 

10 Pc 1 output from the edge direction processing unit 5 is 

generated for each pixel of the output image data D2 on a 
line extending in the edge gradient direction vl in 
accordance with a detection result of the edge detection 
unit 2, processing of edge enhancement is performed by 

15 filtering processing of the interpolated data Pec in the 
edge gradient direction. Thus, a characteristic for the 
filtering processing is variously set, and edge enhancement 
with a desired characteristic can thus be performed. 

In addition, a series of processing is performed such 

20 that a sampling pitch in the output image data D2 is 

different from a sampling pitch in the input image data Dl, 
and sampling pitches are changed together in smoothing 
processing and edge enhancement processing. The 
interpolation processing unit 8 converts the input image 

25 data Dl into the interpolated image data Sll. The weighting 
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coefficient (3 for blending is changed in accordance with the 
reliability of the edge in the edge direction v2 . Weighting 
addition of the image data S3 output from the edge gradient 
direction processing unit and the interpolated image data 
5 Sll is performed in accordance with the weighting 

coefficient P for blending, and the output image data D2 is 
output. Accordingly, excessive edge enhancement in a 
natural image can be effectively prevented. 
(4) Other Embodiments 

10 Although a case where a smoothing processing range in 

an edge direction is changed in accordance with a change of 
the number of taps has been described in the foregoing 
embodiment, the present invention is not limited to this. 
The number of taps may be equally changed in accordance with 

15 a change in a weighting coefficient for smoothing processing. 
In addition, although a case where a smoothing 
processing range and weighting addition performed in the 
blend processing unit are changed in accordance with the 
ratio of the eigenvalue X2 to the eigenvalue A,l and the 

2 0 eigenvalue Xl has been described in the foregoing embodiment, 
the present invention is not limited to this. When a 
practically sufficient characteristic can be ensured, such 
changes may be made in accordance with only the ratio of the 
eigenvalue X2 to the eigenvalue Xl . In addition, instead of 

25 detection of the reliability of the edge in accordance with 
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the ratio of the eigenvalue X2 to the eigenvalue ^1 and the 
eigenvalue Xl , various edge detection methods may be applied. 
In addition, since excessive edge correction for weighting 
addition performed by the blend processing unit occurs in a 
5 point, for example, in which leaves of a tree grows close 
together, in a natural image, such changes may be made in 
accordance with, for example, a characteristic of each 
portion of the image detected as a frequency characteristic 
or the like. 

10 In addition, although a case where a smoothing 

processing range and weighting addition performed by the 
blend processing unit are changed has been described in the 
foregoing embodiment, the present invention is not limited 
to this. When a practically sufficient characteristic can 

15 be ensured, such processing may be omitted. Moreover, one 

of such processing or both of such processing may be omitted 
depending on the type of an image . 

In addition, although a case where an image is 
magnified or reduced by changing a sampling pitch in the 

2 0 input image data Dl has been described in the foregoing 

embodiment, the present invention is not limited to this. 
The present invention is also widely applicable to 
processing for edge correction. In this case, in the edge 
direction processing unit and the edge gradient direction 

25 processing unit, a series of processing is performed in 
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accordance with a sampling pitch of output image data that 
is equal to a sampling pitch of the input image data. In 
addition, the configuration for interpolation processing is 
omitted, and the blend processing unit 10 performs weighting 
addition of output data of the edge gradient direction 
processing unit 7 and the input image data. Thus, edge 
enhancement of input image data can be simply performed. 

In addition, although a case where arithmetic 
processing means performs a predetermined program to process 
image data has been described in the foregoing embodiment, 
the present invention is not limited to this. The present 
invention is also widely applicable to a case where image 
data is processed with hardware configuration. 

Industrial Applicability 
The present invention is applicable, for example, to 
resolution conversion. 



